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ABSTRACT 

Although organizations spend increasingly large 
amounts of money on training, little effort or money is spent on 
evaluating the effectiveness of training programs. When evaluations 
are done, they are often designed for specific programs and often 
consider only participant reactions and immediate learning, and 
ignore behavior changes and long-term results. A study used personal 
interviews to evaluate the effectiveness of an oral communication 
skills training program developed in partnership with a local 
university and implemented at a large medical organization in a 
medium-sized southwestern city. Results indicated that the 23 
participants reported using the skills in the workplace and also 
reported perceiving positive results from using the skills. 
Participants also voluntarily reported success in using the skills 
outside of the workplace, and supplied voluntary comments about how 
the training could be improved, providing a rich source of 
unanticipated data concerning the training program. (Contains 30 
references and two tables of data.) (RS) 
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Evaluation of Communication Training Effectiveness in the Workplace: 

A Case Study 



Abstract 



Although organizations spend increasingly large amounts of money on training, there is 
little effort or money spent on evaluating the effectiveness of training programs. When evaluations 
are done, they are often designed for specific programs and often consider only participant 
reactions and immediate learning, and ignore behavior changes and long-term results. This study 
used personal interviews to evaluate the effectiveness of an oral communication skills training 
program and found that participants reported using the skills in the workplace and also reported 
perceiving positive results from using the skills. Participants also voluntarily reported success in 
using the skills outside of the workplace, and supplied voluntary comments about how the training 
could be improved, providing a rich source of unanticipated data concerning the training program. 
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Evaluation of Communication Training Effectiveness in the Workplace: 

A Case Study 

Introduction 

By any standard, training in organizations is big business today. Estimates of the amount 
of money spent by organizations on training as far back as 1985 ranged from $40 billion per year 
(DeWine, 1987) to $215 billion (Johnson & Kusmierek, 1987a). In addition, estimates are that 
businesses train as many people as the total enrollment in all of America's four-year colleges and 
universities (DeWine, 1987). Despite this enormous investment of both human and fiscal 
resources, there is little evidence that organizations have any formal programs for evaluating their 
return on investment (ROI) for any of these training programs. This is as much a problem for 
organizational communication scholars as for businesses, DeWine (1987) claims, because "For 
scholarship in organizational communication to continue to provide meaningful insights into 
communication behaviors of organizational members, it must provide the means for organizations 
to measure the impact of training on the organization" (p. 123). She continues, "Training 
evaluation is critical to the successful application of organizational communication theory and 
principles. Currently organizations are evaluating the reactions of participants but doing little to 
assess the impact of communication training on other levels of Kirkpatrick's hierarchy" (DeWine, 
1987, p. 121). Kirkpatrick's four levels of evaluation are defined as: (1) reaction, or participants' 
immediate reaction to training and trainer; (2) learning, or whether participants gained the intended 
knowledge or skills in the training environment; (3) behavior, or whether the learning transferred 
to the work environment and persisted; and (4) results, or what benefits accrued to the organization 
as a result of the training. These levels are discussed in detail by Goldhaber (1990) and Goldstein 
(1993), who agree that evaluation is a key part of any training program, and Goldhaber endorses 
"Kirkpatrick's system as feasible for assessing the impact of communication training programs" 
(p. 407). Given this agreement on the need for evaluation, we must look to the reasons for its 
absence as an integral part of organizational training programs. 
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Goldstein (1993) defines evaluation as "the systematic collection of descriptive and 
judgmental information necessary to make effective training decisions related to the selection, 
adoption, value, and modification of various instructional activities" (p. 147). This definition fits 
nicely with the criticism above in that it specifies the reasons that organizations and organizational 
communication scholars need to take an interest in evaluation. Without specific criteria for 
evaluation, neither organizations nor communication scholars have any basis for selecting, 
adopting, valuing or modifying various training programs. In discussing the criteria for 
evaluation, Goldstein (1993) notes that there are both process measures, defined as "measures that 
examine what happens during instruction" (p. 170), and outcome measures, which are simply 
measures of observable results. He claims that a reliance on outcome measures alone can yield 
invalid evaluation of the effectiveness of a training program because of possible intervening 
variables. Thus he advocates an approach which combines the use of process and outcome 
measures. But just as he advocates a set of information to be collected, he defines the problem 
facing those who would develop methods for evaluation. 

One of the biggest problems in evaluating organizational training is in deciding what 
information to collect and how to use it to evaluate training effectiveness. It is relatively easy to 
assess whether the training was enjoyable through the use of a simple survey conducted at the end 
of class (Kirkpatrick's level one, reaction), and this is often the only evaluation conducted for 
many training programs. It is also a fairly straightforward process to ascertain whether or not 
participants learned what they were supposed to learn through the use of observation or pre- and 
post-test instruments administered at the end of the training (level two, learning), and this is done 
often, although less frequently than reaction evaluation. The difficulties appear when we try to 
assess either training transfer or behavior changes (level three, behavior) or the long-term benefits 
of training (level four, results), both to the employees and to the organization. One of the main 
reasons for these difficulties is that the effects of training must be assessed or evaluated in an 
environment which has numerous other forces which affect employees and their interactions with 
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others. It is difficult to determine which observed effects are due to the training and which are due 
to other forces. 

While there are some ways to minimize the effects of these intervening or moderating 
variables on an evaluation, many trainers are not willing to invest the necessary time. This is 
unfortunate, since effective evaluations are key to revising and modifying training courses to 
ensure that they achieve the desired results. In addition, agreement is necessary on methods and 
instruments to allow comparability among programs. A review of the literature will provide an 
overview of what has been done and what remains to be done. 

Literature Review 

The literature abounds with reports of poor or non-existent evaluation programs (Cordcr, 
1988; Johnson & Kusmierek, 1987a and 1987b; Mattox & Smith, 1983), calls for more evaluation 
(Goldhaber, 1990; Goldstein, 1993), pleas for standardization of evaluation methods (DeWine, 
1987; Stevens & Hellweg, 1990), and reports of ineffective training programs (Baum & James, 
1984; Elmes & Costello, 1992). Meanwhile, reports of successful training programs focus more 
on method of training and evaluation of specific programs (Anderson & Sharpe, 1991; Burnett & 
Thompson, 1986; Dickson, 1989; Evans, Stanley, Mestrovic, & Rose, 1991; Gray, Buerkel- 
Rothfuss, & Yerby, 1986; Hudson, Doyle, & Venezia, 1991; Kurtz, 1982; Morrow & Har^ie, 
1986; Papa & Graham, 1991; Rybczyk, 1987; Sellick, 1991) and less on general effectiveness, 
despite the widespread belief in the benefit of such programs (Bell & Kerr, 198 7 ; Armstrong, et 
al M 1986), and evaluators have not settled on one of the standardized instruments for evaluation. 

Reasons for Lack of Evaluation 

Among the reasons cited for lack of evaluation, management attitudes account for a 
significant portion. Management is often unwilling to spend the money necessary for effective 
evaluation (Goldstein, 1993). Lack of understanding of the importance of evaluation contributes to 
this unwillingness, and is cited by Bell and Kerr (1987) and by Goldstein (1993), while Bell and 
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Kerr also report that management believes that training programs which are selected and 
administered must be effective and therefore sees no need for evaluation. This inherent belief is 
similar to the attitude found by Armstrong, et al., (1986), who surveyed nurses, dentists, and 
doctors and found that most considered communication training necessary and valuable at several 
educational levels. 

In addition, Corder (1988) and Johnson and Kusmierek (1987a, 1987b) found that training 
directors and Human Resource (HR) consultants and managers believed in evaluation and did 
perform some form of evaluation, but did not evaluate training programs in sufficient detail to 
assess results or behavioral changes. Corder (1988) reported that there were no baseline 
measurements established to allow for assessment of changes due to programs. Finally, Stevens 
and Hellweg (1990) believe that the variety of instruments and guidelines contributes to the lack of 
ability to compare programs in meaningful ways. 

Recommendations for Evaluation 

Recognizing the problems caused by lack of training program evaluation, several 
researchers have proposed methods to address the issue. Stevens and Hellweg (1990) call for 
standards of evaluation which allow for comparison of results across methodologies and training 
formats, and they also call for investigation of the appropriate time frame for evaluating results. 
They acknowledge De Wine's (1987) Training Impact Questionnaire (TIQ) as a useful contribution 
to assessing reaction and behavior, levels one and three of Kirkpatrick's hierarchy (Goldhaber, 
1990; Goldstein, 1993), and call for additional validation of this instrument. DeWine (1987) 
presents the TIQ as a validated instrument for assessing reaction and behavior change concerning a 
single skill through self-report data. 

Goldhaber (1990) presents a candidate method of evaluation which addresses all four levels 
of Kirkpatrick's hierarchy and offers an example of its use. Similarly, Bell and Kerr (1987) 
suggest methods for evaluating each of four levels which resemble Kirkpatrick's levels, and they 
also present a practical example of their use. The International Communication Association has 
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developed the Communication Audit (Goldhaber & Rogers* 1979) which can be used to assess 
various elements of communication in organizations, and changes in communication elements can 
be effective indicators of changes which may be due to training. McLean (1988) argues for the use 
of surveys to assess communication climate (Falcione, Sussman, & Herden, 1987) and presents a 
proposed format and a series of questions which could be used to construct a customized climate 
Survey. He argues that a customized survey will be better able to address organizational goals than 
a general survey. 

Finally, several researchers note limitations and pitfalls in methods of evaluation. Johnson 
and Kusmierek (1987b) discuss self-report methods and note that "The limitations of self-report 
data have been noted by many researchers and careful attention should be paid to ensure that these 
limitations are not overlooked when interpreting evaluation data" (Johnson & Kusmierek, 1987b, 
p. 155). One of these limitations was discussed by Sprangers and Hoogstraten (1989) who found 
that response shift bias was a factor that needed to be considered when administering evaluation 
instruments which relied on self-report data prior to training. Response shift is a change in a 
participant's internal standard for determining his or her level of functioning on a given dimension. 
Response shift bias can be prevented by administering a behavioral pretest which gives the 
participant a more realistic standard for personal assessment and reduces the potential for response 
shift, thus allowing comparability between pre- and post-training self-report data. Johnson and 
Kusmierek ( 1987b) also warn evaluators to be sure they understand what it is that they are 
evaluating. They claim that "Many HR trainers confuse the results of implementation evaluations 
with instructional and job performance effects.. . .It is gratuitous to assume a positive relationship 
between trainees' or trainers' level of satisfaction and instructional and job performance effects.. .In 
essence, HR trainers are hypothesizing a causal relationship between participation in a training 
program and instructional effects without adequately testing the relationship" (Johnson & 
Kusmierek, 1987b, pp. 155-156). 



ERLC 



8 



COMM TRAINING EVALUATION 8 



Summary 

As can be seen from the review above, there are standardized instruments available for 
evaluation, but none has gained the unified support of evaluators. Because needs assessment is a 
key part of the design of any effective training program (Goldhaber, 1990; Goldstein, 1993; 
Johnson & Kusmierek, 1987a and 1987b), and because needs assessment will be combined with a 
review of organizational goals and current knowledge, skills and abilities (KSAs) before program 
implementation (Goldstein, 1993), it may be nearly impossible to develop standard methods of 
evaluation as called for by Stevens and Hellweg (1990). In fact, there are arguments for evaluation 
criteria being built in as the programs are being developed, which implies different criteria for each 
program, and there are also suggestions that customized surveys are the most appropriate methods 
for evaluation (McLean, 1988). The most important thing seems to be at present that evaluation be 
done at a level beyond the simple reaction sheets (level one) and the occasional pre- and post-test 
administered immediately after the class (level two). Until adoption of standard methods and 
instruments, it is important to use a method which will yield the data necessary for an effective 
evaluation. In addition, evaluation at other levels should attempt to establish a relation between the 
levels, such as learning causing behavior or behavior causing results. This suggests the following 
set of research questions. 

Research Questions 

RQ1 : How does formal communication training in oral communication skills affect oral 
communication behaviors? 

This question looks for changes in behavior reflecting transfer of the training or 
learning to the work environment or to another environment. This transfer will 
correspond to Kirkpatrick's third level of evaluation, behavior (Goldhaber, 1990; 
Goldstein, 1993). 



ERLC 



9 



COMM TRAINING EVALUATION 9 



RQ2: Do changes in oral communication behaviors yield positive results for individuals? 
This question looks for results of behavior changes potentially identified in RQ1 to 
see whether these results are positive for the individual. Since the results will not be 
evaluated for the organization, this question only partially addresses Kirkpatrick's fourth 
level of evaluation, results (Goldhaber, 1990; Goldstein, 1993). 

Theoretical Perspective 

A psychological perspective (Krone, Jablin, & Putnam, 1987) will be used to determine 
individual perceptions of the various results of the communication skills training. Self-report data 
of perceived changes in behavior due to the training, and self-report data of perceived benefit, if 
any, of changed behaviors will be gathered. Once the data from this study are analyzed, it should 
be possible to develop a more rigorous quantitative approach to assess behavioral changes and 
results throughout the organization. 

Study Design and Methodology 

For the past several years, a large medical organization in a medium-sized southwestern 
city, in partnership with the local university, has been providing training in basic communication 
skills to its front-line employees. The work has been done under a grant from the Department of 
Education specifically designed to bring training in workplace literacy and basic skills to front-line 
employees. The specific courses that have been taught include oral communication skills, written 
communication skills, reading and study skills for employees anticipating returning to school or 
attempting their GEDs, effective presentations, and communication for supervisors, designed for 
new or soon-to-be supervisors or team leaders. 

While evaluation was a key part of the grant proposal for the project (Bernhardt & Meyer, 
1991), only one attempt has been made to do more than evaluate immediately whether participants 
liked the course and whether they learned what they were supposed to learn (Ham & Acchione- 
Noel, 1991). There is a certain amount of anecdotal evidence concerning long-term results, but 
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there has been virtually no formal follow-up of behavioral changes and results. This organization 
provides an excellent setting for evaluating the effect of communication skills training over a long 
period of time. 

Description of Organization Studied 

As noted above, this is a large medical organization run by a Board of Directors, a 
President and Chief Executive Officer, and six administrative vice-presidents. Personnel at the 
organization range from highly skilled professionals with staff positions to support personnel with 
minimal education who are often minorities and women, many of them with English as a second 
language. The departments range from those that deal almost exclusively with customers and the 
public to those who provide support services and deal only with internal communications. The 
range of education levels and communication skills within the organization is extremely broad, and 
the requirement for dealing with a broad cross-section of the public adds an additional burden to 
the communication demands on all employees, especially front-line employees. The 
communication skills training program described above was developed to address the needs of 
these employees specifically. 

Description of Participants and Sampling Method 

Communication skills training has been presented in this organization since late 1990. The 
oral communication skills training course was selected for evaluation for this study because of the 
critical nature of oral communication in this organization, and attendance lists for that course were 
obtained from the Educational Services Department to serve as the basis for sample selection. A 
purposive sample of participants, consisting of all members of three departments who had 
completed the course and who were still employed by the organization, was chosen for interviews. 
The three departments were selected because a large percentage of each department had attended the 
course during a short time frame, and time frames for each department had been separated by large 
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periods of time, nine months to one year. This provided a basis for evaluating similar groups, and 
for evaluating the training itself at three specific time points. 

Participants from Department A completed the course during April-May 1992. All four , 
participants who completed the course were interviewed, along with a supervisor who had not 
completed the course, but who had observed the participants before and after the training. 
Participants from Department B completed the course during April- June 1991. Seven of the nine 
participants who completed the course were still employed and were interviewed. Participants 
from Department C completed the course during October-December 1990. In this department, all 
members had completed the course in 1990 by management direction. Of the twenty-four original 
participants, eleven were still employed and were interviewed. This gave a total of twenty- two 
participants and one supervisor who were interviewed for this study. 

Conceptual and Operational Definitions of Theoretical Concepts 

Operational definitions for the concepts in the research questions are provided below. 

• Communication training is defined as a process that provides for presentation, 
demonstration, and practice of communication skills or behaviors. 

• Communication skills are defined as: 

Active listening; 

Paraphrasing to clarify or understand the situation; 

Dealing with feelings or emotions, both personal and those of others; 

Empathizing; 

Being aware of the different communication styles among different people, 

such as task-oriented or people-oriented styles; 

Body language and non-verbal communication; and 

Discussing and controlling conversations. 
(N. B. - The skills listed above overlap to some extent, such as active listening being an 
integral part of both empathizing and paraphrasing. They are listed this way because they 
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are taught this way in the oral communication skills course, with paraphrasing and 
empathizing building on active listening.) 

♦ Changes in communication behaviors are defined as voluntary statements of perceived 
increased or decreased use of the communication skills above. 

♦ Positive results are defined as voluntary statements of examples of perceived improvement 
in an individual's interaction with his or her environment. Improvement may come in the 
areas of communication satisfaction, reduced tension in the workplace, increased comfort 
or reduced anxiety or communication apprehension in oral communication, better job 
evaluations or recognition from peers or supervisors, or similar outcomes. 

Note that both changes in communication behavior and positive results are defined using 
voluntary self-report data. Because no baseline data exists for the study, and because a 
psychological perspective places the locus of analysis in individual perception and conceptual 
filters (Krone, Jablin, & Putnam, 1987), this is deemed to be an appropriate method for this stage 
of evaluation. 

Measures Used and Rationale 

In order to gather self-report data from participants concerning changes in behavior and 
perceived benefits, a scripted interview form was developed. The scripted interview used by Ham 
and Acchione-Noel (1991) was used as an example of a proven technique for evaluating the written 
communication training in this organization. Questions were arranged in a funnel order, asking 
participants first to remember anything they could about the course and then asking for more detail 
as they provided more detailed information. If specific skills were recalled or described, additional 
questions were listed about a specific incident of using that skill or what the result had been. In no 
case were specific skills listed in the questions or asked about in the follow-up or probing 
questions in order to avoid leading or biasing the interviewees as much as possible. Finally, the 
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questions were reviewed by the training program project directors for any additional comments or 
suggestions. 

Procedure 

Supervisors in each of the departments were contacted and the project was explained to 
them, noting that it had the approval of the Chief Operating Officer and the Vice president of 
Human Resources. Supervisors' approval was obtained for interviewing their employees, and 
then individual employees were contacted and interview times were scheduled. Supervisors were 
very cooperative and allowed employees to be interviewed on work time. Interviews were 
conducted in various sites, including the cafeteria or empty offices or conference rooms, in order to 
ensure privacy away from the immediate work environment. 

The purpose of the project was explained to each participant and they were asked for 
permission to audiotape the inten ;ew. All agreed to this and all interviews were audiotaped. 
Interviews took anywhere from 15 minutes to 45 minutes. The scripted interview was followed to 
the extent possible, although at times participants volunteered answers to questions out of order 
and these were followed up at that time. At no time did the interviewer mention any of the skills or 
potential benefits unless they had been explicitly mentioned by the participant first. Probing 
questions asked simply for elaboration or for an example of using a skill already mentioned. 

Partial transcriptions were made of all tapes for ease of reference, and these were examined 
for explicit reference to any of the skills listed above, as well as for descriptions of use of skills 
without actually naming the skill. Repeated themes or reports of benefits of using the skills were 
marked for later analysis and grouping. 

Results 

General Findings 

Of the twenty-three participants who were interviewed, all but one reported either 
observing or experiencing some changes in behavior and positive results attributed to the training. 
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Incidents mentioned by one participant were often confirmed by other participants, leading to 
confidence in the validity of the responses. Participants often could not name the skills that were 
taught in the course, but they then went on to describe the use of skills later in the interview. This 
was particularly apparent in Department C which had completed the course in late 1990. Seven 
participants volunteered the observation that they used the skills with children or with other family 
members, providing a broader definition of environment than simply the workplace. Finally, six 
participants volunteered the observation they used the skills automatically, without consciously 
thinking about them. 

Specific Findings for Research Questions 

Research Question 1 asked how communication training affected communication 
behaviors. The responses are shown in Table 1 as the number of participants who reported using a 
particular skill or communication behavior. Based on the results presented in Table 1, the answer 
to Research Question 1 is that participants report perceiving that communication training results in 
changes to oral communication behaviors, and these changes persist over a period of nine months 
to several years, at least for some participants. 



Skills Reported as Used by Participants 


Number Who Reported 
Using Skill (n=23) 


Active listening 


12 


Paraphrasing to clarify or understand 


5 


Dealing with feelings and emotions 


4 


Empathizing 


3 


Communication styles 


2 


Non-verbal and body language 


2 


Discussing and controlling conversations 


2 



Table 1 . Perceived Behavior Changes By Skill as Reported by Participants 



Research Question 2 asked whether the changes in oral communication behaviors yielded 
positive results for the participants. The responses are shown in Table 2 as the number of 
participants who reported perceiving each result. Results are shown as repeated themes which 
appeared in the interviews. Based on the data shown in Table 2, the answer to Research Question 
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2 is that participants report perceiving that changes in communication behaviors attributed to 
communication training yield positive results for individuals. 



Positive Results Reported by Participants as 
Repeated Themes from Interviews 


Number Who 
Reported (n=23) 


Bettergroup communication and less tension 
Improved family and personal communication 
More teamwork and smoother work flow 
Positive attitude change toward job, work area, coworkers 
Reduced anxiety and less communication apprehension 
Miscellaneous (Stress reduction, recognition from 
supervisor or peers, increased self-esteem) 


9 
7 
5 
5 
3 
6 



Table 2. Perceived Positive Results Due to Changes in Communication Behaviors 



Patterns and Contradictions Among Data Sources 

The patterns observed in the interviews were remarkably consistent, with the exception of 
the single individual who reported no benefits from the program. This individual stated that when 
he was involved in working he was not "thinking about using skills from some communication 
class he had taken." This seemed to reflect a perception that he had to think about the skills to use 
them, and it contradicts the reports from at least six participants that they use the skills 
automatically. 

Discussion 

The results presented above support and extend the literature on communication training 
and evaluation in several ways. First, they support the results of several studies showing that 
participants perceive that communication training can change communication behavio; s and that 
these changes in behavior can persist over time (Mattox & Smith, 1983; Papa & Graham, 1991). 
Second, in their reliance on self-report data using a psychological perspective, they confirm that 
participants will perceive changes in behavior based on the skills taught in class and report them 
consistently to researchers. This finding supports De Wine's (1987) use of self-report data on the 
Training Impact Questionnaire, a method which was questioned by Stevens and Hellweg (1990). 
Finally, it extends the current literature by providing one of the few studies which looks at both 
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behavior change (level three of Kirkpatrick's evaluation hierarchy) and results of those behavior 
changes (level four) in a single study. Results were found to be positive for the employees, who 
could report specific benefits from the use of the communication skills. Results for the 
organization were implied as they were by Bell and Kerr (1987), who assumed that more efficient 
use of time, better communication, and fewer errors would result in better productivity for 
employees who had been trained. Whether or not this implication is valid is a subject for future 
research. 

The study also demonstrated the power of using personal interviews to gather evaluation 
data of this type. In fact, while it is possible to gather anecdotal data and personal success stories 
using other methods, personal interviews provide an excellent means for doing so. For example, 
one participant reported seeking out difficult customers to work with, and he reported great success 
in doing so. Another reported understanding himself and his response to stress better after the 
course. This led him to request a change in job assignment rotations so as to balance his work in 
stressful areas, and he reported being much happier since the change. Still a third reported 
increased self-esteem and confidence in communication which led her to apply for a different 
position. Her supervisor recognized her new skills and supported her application, and she was 
given the job. It may have been possible to obtain this data using open-ended questions on a 
general survey, but it is less likely, and personal interviews should continue to supplement other 
means of data gathering. 

Several implications for communication training design and development also emerged 
from this study. First, there was a confirmation of the importance of role-playing (Hudson, 
Doyle, & Venezia, 1991) during the training, and second, confirmation of the importance of 
practice and feedback (Kurtz, 1982; Rybczyk, 1987) when learning new skills and transferring 
them to a new environment. Role-playing was a significant part of the course structure for the oral 
communication course, and it was one of the features of the course that was mentioned by many 
participants as being very helpful, along with the importance of the role plays using real-life 
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situations to improve relevance to the transfer environment. This suggests that role-playing should 
continue to be a major part of communication training. 

The importance of practice and feedback emerged in comments by participants in the two 
larger departments. Participants reported using the skills with each other during and after the class 
in what might be called a manner of joking self-awareness. They would recognize when one or 
another was trying to use or practice a skill and would comment on it. As skills began to be used 
more naturally, this self-awareness diminished, but it was very important as a source of 
reinforcement using practice and feedback immediately following the class. The participants in the 
smallest department did not report this awareness and practice, and at least one participant thought 
that the course would have been more effective if more people from the department had taken it. 
This suggests two implications for training design and delivery. First, to the extent possible, large 
or intact groups should^be trained or encouraged to participate concurrently. This will facilitate the 
use of the practice and feedback which was so important to changing the communication behaviors 
of participants. Second, if possible, follow-up sessions should be scheduled for participants 
several months after the initial training. This was suggested by several participants and would 
provide for intensive practice and feedback after a period of using the skills and incorporating them 
into a repertoire. It could also provide an opportunity to use actual situations in role plays, and 
could provide an excellent setting for evaluation of behavior changes by trainers. 

Another important finding was confirmation of the importance of the classroom or 
workshop experience itself as reported by Elmes and Costello (1992). One of the reasons for 
Department C attending the class as a group was a perception by management that communication 
had broken down in that department and that it was affecting work. Many participants reported 
that one of the primary benefits of the class was simply having a safe place to discuss problems 
and to work out communication problems with coworkers and supervisors without fear of 
retribution. In this case, the trainers acted as facilitators to allow the problems to surface and be 
discussed. As Elmes and Costello (1992) note, "Effective communication skills training begins 
when people are encouraged and given the opportunity to relate to one another in meaningful ways. 
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It begins when people have a chance to explore and ask questions about relationships and their 
contextual influences" (p. 443). Several participants commented that it would be good to do that 
again and to incorporate the new people in the department, a suggestion that may be worth 
pursuing. 

Finally, this study demonstrated that it is possible to gather meaningful evaluation data 
without the use of surveys and quantitative methods. While the results cannot be quantified into 
bottom-line figures for the organization, they do provide an indication that participants consider the 
training beneficial and useful in many ways. The results have good internal validity in that they 
were collected across three departments having very different responsibilities, and given the 
uniformity of results, one could anticipate good intra-organization validity. External validity 
beyond this organization is problematical, although it is possible that the results may be 
generalizable to similar organizations using the same program and trainers. This will be confirmed 
during continuation of this study in several of those organizations using the training in the future. 

Strengths and Limitations of Current Study 

Some of the strengths and limitations of the current study have been addressed above. 
Confirmation and extension of the current literature is obviously one of the strengths, as is the 
demonstration of connecting evaluation levels three and four in a single study. The implications 
drawn for training design and delivery should provide important benefit for trainers and training 
designers. Finally, confirmation that communication training evaluation is possible through the 
use of personal interviews should help to convince management of the benefits of continuing to 
perform such evaluations. 

Obvious limitations are the lack of generalizable methods for evaluation as called for by 
Stevens and Hellweg ( 1 990) and the lack of development of baseline data called for by Corder 
(1988). The external validity or generalizability of the results may be a limitation, but this will be 
explored in future research in this organization and others using the same training program. 
Finally, the study limited itself to self-report data from participants only, with the exception of one 
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supervisor, and it could benefit from triangulation of observations by coworkers and other 
supervisors. While there was some confirmation of data by cross-report by participants, it was 
limited and informal. This should be remedied in the continuing studies planned in the future. 

Conclusions 

This study found that, in this organization, participants in formal communication training 
perceive behavioral changes which they attribute to the training, and these behavioral changes 
persist over time. Further, these participants report positive results from the use of the changed 
behaviors, thus establishing a perceived positive effect due to the communication training program. 
This suggests that the decision to use communication training in this organization was worthwhile, 
especially for those employees who participated. Given the uniformity of responses across the 
three departments examined, one could expect good intra-organizational validity for this training, 
meaning that a decision to continue the training could be expected to produce similar favorable 
results in other departments. Given that similar organizations exist in other cities, it is possible that 
the training results may generalize to these other organizations, but this study did not address that 
issue. 

The study also demonstrated that it is possible to perform useful evaluations of levels other 
than reaction and learning even in the absence of baseline performance data and without using 
extensive quantitative methods. In fact, the psychological perspective and the personal interviews 
yielded results which might not have been obtainable with a general survey. The implications for 
training design and delivery would not have appeared with a survey which simply examined 
outcome variables, lending credence to Goldstein's (1993) observation concerning the potential 
error of not including process variables in evaluation. It is also unlikely that the rich supply of 
anecdotal evidence would have appeared using a general survey. 

While the results do not provide the standardization called for by Stevens and Hellweg 
(1990), they do provide evidence of the value of evaluating communication training programs at 
the behavior and results levels. This is a valuable addition to the current literature, which contains 
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few published reports of evaluations combining these levels. Continued research along these lines 
should continue to provide valuable data for researchers in organizational communication, who 
must address these issues to stay abreast of current trends in rapidly changing organizations. 
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